A decoder CNN is then used to combine the content features of one frame and pose features of a different frame to reconstruct the frame that corresponds to the pose features. This paper presents a new loss for the disentanglement of pose features from input images using a discriminator network (similar to adversarial loss). They train a discriminator network find similarities between features from the pose encoder of images from the same video but different time steps. The pose encoder is then trained to find features that the discriminator cannot distinguish as being similar. At convergence, the pose encoder features will only contain difference between the frames which should reflect the pose change of the person in time. After the encoder-decoder CNN has been trained to identify the content and pose features, an LSTM is trained to take the content features of the last frame and previous pose features to predict the next pose features. The disentangled content and pose features result in high quality video prediction. The experimental results backup the advantage of this method in terms of feature quality for classification tasks, and shows quantitative and qualitative performance boost over the state-of-the-art in video prediction.

Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems in computer vision, medical imaging, etc. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance. Inspired by recent theoretical understanding on generalizability, expressivity and optimization landscape of neural networks, as well as the theory of convolutional framelets, here we provide a unified theoretical framework that leads to a better understanding of geometry of encoder-decoder CNNs. Our unified mathematical framework shows that encoder-decoder CNN architecture is closely related to nonlinear basis representation using combinatorial convolution frames, whose expressibility increases exponentially with the network depth. We also demonstrate the importance of skipped connection in terms of expressibility, and optimization landscape.